Performing software testing with best possible user experience

ABSTRACT

A computer-implemented method, system and computer program product for testing a software application. User interactions with the functional and non-functional aspects of software applications are monitored. User experience profiles are then generated for the monitored users based on the monitored interactions. The best user experiences in connection with interacting with specific functional and non-functional aspects of the software applications are then identified based on the generated user experience profiles. Virtual users having different types of best user experiences in connection with interacting with different functional and non-functional aspects of the software applications are created. The software application is then tested using the virtual users to determine if the functional and non-functional aspects of the software application satisfy the best user experiences associated with the virtual users. A message may be generated indicating which functional and/or non-functional aspects of the software application did not satisfy a virtual user&#39;s best user experience.

TECHNICAL FIELD

The present disclosure relates generally to software testing, and moreparticularly to performing software testing with the best possible userexperience, such as during the testing stage of the software productdevelopment process prior to being deployed in production.

BACKGROUND

Software testing, such as during the testing stage of the softwareproduct development process, is an investigation conducted to providestakeholders with information about the quality of the software productor service under test. Software testing can also provide an objective,independent view of the software to allow the business to appreciate andunderstand the risks of software implementation. Test techniques includethe process of executing a program or application with the intent offinding failures and verifying that the software product is fit for use.

SUMMARY

In one embodiment of the present disclosure, a computer-implementedmethod for testing a software application comprises identifying bestuser experiences in connection with interacting with specific functionaland non-functional aspects of software applications. The method furthercomprises creating virtual users having different types of best userexperiences, which are obtained from the identified best userexperiences, in connection with interacting with different functionaland non-functional aspects of the software applications. The methodadditionally comprises performing testing of the software applicationusing the virtual users to determine if functional and non-functionalaspects of the software application satisfy the best user experiencesassociated with the virtual users.

Other forms of the embodiment of the computer-implemented methoddescribed above are in a system and in a computer program product.

The foregoing has outlined rather generally the features and technicaladvantages of one or more embodiments of the present disclosure in orderthat the detailed description of the present disclosure that follows maybe better understood. Additional features and advantages of the presentdisclosure will be described hereinafter which may form the subject ofthe claims of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present disclosure can be obtained whenthe following detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 illustrates a communication system for practicing the principlesof the present disclosure in accordance with an embodiment of thepresent disclosure;

FIG. 2 is a diagram of the software components used by the testingsystem to perform testing of a software application using the users'best experiences in connection with the functional and non-functionalaspects of the software application under test in accordance with anembodiment of the present disclosure;

FIG. 3 illustrates an embodiment of the present disclosure of thehardware configuration of the testing system which is representative ofa hardware environment for practicing the present disclosure;

FIG. 4 is a flowchart of a method for testing a software application todetermine if it satisfies the users' best experiences with respect tothe functional and non-functional aspects of the software application inaccordance with an embodiment of the present disclosure; and

FIG. 5 illustrates an exemplary user experience profile containinginformation pertaining to the user's experience in connection with thefunctional and/or non-functional aspects of the software application inaccordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

As stated in the Background section, software testing, such as duringthe testing stage of the software product development process, is aninvestigation conducted to provide stakeholders with information aboutthe quality of the software product or service under test. Softwaretesting can also provide an objective, independent view of the softwareto allow the business to appreciate and understand the risks of softwareimplementation. Test techniques include the process of executing aprogram or application with the intent of finding failures and verifyingthat the software product is fit for use.

Software testing involves the execution of a software component orsystem component to evaluate one or more properties of interest. Ingeneral, these properties indicate the extent to which the component orsystem under test: meets the requirements that guided its design anddevelopment, responds correctly to all kinds of inputs, performs itsfunctions within an acceptable time, is sufficiently usable, can beinstalled and run in its intended environments and achieves the generalresult its stakeholders desire.

However, such software testing fails to quantify the users' bestexperiences with the functional and non-functional aspects of thesoftware application under test. For example, when a user gets adrop-down list populated within a millisecond, the user may expect suchan experience in populating another drop-down list in the same or in adifferent application.

By not testing software applications to determine if the functional andnon-functional aspects of the software application satisfy the users'best experiences, such software applications may not be well received bythe targeted users despite performing well during the testing stage ofthe software product development process due to the poor user experienceof the software application.

Unfortunately, there is not currently a means for testing softwareapplications with the users' best experiences in connection with thefunctional and non-functional aspects of the software application undertest.

The embodiments of the present disclosure provide a means for performingsoftware testing that includes testing the software application todetermine if it satisfies the users' best experiences with respect tothe functional and non-functional aspects of the software applicationunder test.

In some embodiments of the present disclosure, the present disclosurecomprises a computer-implemented method, system and computer programproduct for testing a software application. In one embodiment of thepresent disclosure, user interactions with the functional andnon-functional aspects of the software applications are monitored.“Functional” aspects of the software applications, as used herein, referto the requirements that the software must perform. For example, suchfunctional aspects may include the following types of requirements:transaction handling, business rules, certification requirements,reporting requirements, administrative functions, authorization levels,audit tracking, external interfaces, etc. “Non-functional” aspects ofthe software applications, as used herein, specify how the softwareapplication achieves such requirements. For example, the non-functionalaspects specify quality attributes of the software, scalability,capacity, maintainability, performance (e.g., response time),portability, security, availability, reliability, recoverability,serviceability, etc. User experience profiles are then generated for themonitored users based on the monitored interactions. A “user experience”profile, as used herein, refers to a profile that includes datapertaining to a user's perception of utility, ease of use and efficiencyin interacting with the functional and non-functional aspects of thesoftware applications. For example, such data may include the user'sexperience in connection with interacting with a window, a menu, anicon, a widget, a table, a cursor, a pointer, an insertion point, etc.The best user experiences in connection with interacting with specificfunctional and non-functional aspects of the software applications arethen identified based on these generated user experience profiles. Forexample, out of the generated user experience profiles, it may bedetermined that a drop-down menu of ten entries was populated at thefastest time of 1 millisecond. As a result, it may be deemed that thebest user experience corresponds to populating a drop-down menu of tenentries at 1 millisecond. A “best user experience,” as used herein,refers to the most effective or desirable interaction with a functionalor non-functional aspect of the software application, such as undergoingan interaction with a functional or non-functional aspect of thesoftware application in the least amount of time. Virtual users havingdifferent types of best user experiences (obtained from the identifiedbest user experiences) in connection with interacting with differentfunctional and non-functional aspects of the software applications arecreated. A “virtual user,” as used herein, refers to a virtualrepresentation of the user whose best user experience will berepresented by the virtual user. “Virtual representation,” as usedherein, refers to symbolizing a real user via the use of the user's bestuser experience that will be used to test the software application undertest by the virtual user. For example, a virtual user is created to testa software application to determine if the software applicationsatisfies the best user experience in connection with the non-functionalaspect of populating a menu of ten items, where such a best userexperience is associated with such a created virtual user. The softwareapplication is then tested using the virtual users to determine if thefunctional and non-functional aspects of the software applicationsatisfy the best user experiences associated with the virtual users. Ifsome of the functional or non-functional aspects of the softwareapplication did not satisfy the best user experiences associated withthe virtual users, then a message may be generated indicating whichfunctional and/or non-functional aspects of the software applicationunder test did not satisfy a virtual user's best user experience. Such amessage may be utilized by the software developer to correct suchdeficiencies. In this manner, software applications may be tested todetermine if the software applications satisfy the users' bestexperiences with respect to the functional and non-functional aspects ofthe software application under test.

In the following description, numerous specific details are set forth toprovide a thorough understanding of the present disclosure. However, itwill be apparent to those skilled in the art that the present disclosuremay be practiced without such specific details. In other instances,well-known circuits have been shown in block diagram form in order notto obscure the present disclosure in unnecessary detail. For the mostpart, details considering timing considerations and the like have beenomitted inasmuch as such details are not necessary to obtain a completeunderstanding of the present disclosure and are within the skills ofpersons of ordinary skill the relevant art.

Referring now to the Figures in detail, FIG. 1 illustrates an embodimentof the present disclosure of a communication system 100 for practicingthe principles of the present disclosure. Communication system 100includes computing devices 101A-101C (identified as “Computing DeviceA,” “Computing Device B,” and “Computing Device C,” respectively, inFIG. 1 ) connected to a software development system 102 via a network103. Computing devices 101A-101C may collectively or individually bereferred to as computing devices 101 or computing device 101,respectively. It is noted that both computing devices 101 and the usersof computing devices 101 may be identified with element number 101.

Computing device 101 may be any type of computing device (e.g., portablecomputing unit, Personal Digital Assistant (PDA), laptop computer,mobile device, tablet personal computer, smartphone, mobile phone,navigation device, gaming unit, desktop computer system, workstation,Internet appliance and the like) configured with the capability ofconnecting to network 103 and consequently communicating with othercomputing devices 101 and software development system 102. It is notedthat both computing device 101 and the user of computing device 101 maybe identified with element number 101.

Software development system 102 is a system utilized, such as bysoftware developers, in the process of creating, designing, deployingand supporting software. Examples of such software development systemsinclude, but not limited to, RAD Studio®, Embold®, Collaborator®, Studio3T®, NetBeans®, Zend Studio®, Microsoft® Expression Studio, etc.

In one embodiment, using software development system 102, softwaredevelopers develop software products to be tested, such as being testedto determine if the software products satisfy the users' bestexperiences with respect to the functional and non-functional aspects ofthe software products.

Network 103 may be, for example, a local area network, a wide areanetwork, a wireless wide area network, a circuit-switched telephonenetwork, a Global System for Mobile Communications (GSM) network, aWireless Application Protocol (WAP) network, a WiFi network, an IEEE802.11 standards network, various combinations thereof, etc. Othernetworks, whose descriptions are omitted here for brevity, may also beused in conjunction with system 100 of FIG. 1 without departing from thescope of the present disclosure.

System 100 further includes a testing system 104 interconnected withcomputing devices 101 and software development system 102 via network103. In one embodiment, testing system 104 is configured to testsoftware applications, such as during the testing stage of the softwareproduct development process, to determine if the software applicationssatisfy the users' best experiences with respect to the functional andnon-functional aspects of the software applications. In one embodiment,such software applications are in the process of being developed bysoftware developers using software development system 102. In oneembodiment, such software applications are accessed from softwaredevelopment system 102 to be tested by testing system 104. In oneembodiment, user interactions with functional and non-functional aspectsof software applications are monitored by testing system 104. In oneembodiment, such information is used to populate “user experienceprofiles” generated by testing system 104. A “user experience” profile,as used herein, refers to a profile that includes data pertaining to auser's perception of utility, ease of use and efficiency in interactingwith the functional and non-functional aspects of the softwareapplications. For example, such data may include the user's experiencein connection with interacting with a window, a menu, an icon, a widget,a table, a cursor, a pointer, an insertion point, etc. Such data mayinclude information, such as the speed in which the table was populated,the number of steps performed in resizing a window or selecting anobject from a menu, the amount of time it takes a user to move thecursor or pointer from one position to another position using usercontrols, such as a mouse, touch pad or similar input device, the amountof time it takes a user to identify the insertion point, such as thelocation at which text is inserted, etc.

In one embodiment, testing system 104 utilizes such user experienceprofiles to identify the best user experience in connection withspecific functional and non-functional aspects of the softwareapplication (e.g., populating a drop-down menu). For example, out of theuser experience profiles examined by testing system 104, it may bedetermined that a drop-down menu of ten entries was populated at thefastest time of 1 millisecond. As a result, it may be deemed that thebest user experience corresponds to populating a drop-down menu of tenentries at 1 millisecond. A “best user experience,” as used herein,refers to the most effective or desirable interaction with a functionalor non-functional aspect of the software application, such as undergoingan interaction with a functional or non-functional aspect of thesoftware application in the least amount of time.

Upon identifying a best user experience in connection with specificfunctional and non-functional aspects of the software application,testing system 104 is further configured to create “virtual users” tocorrespond to a best user experience for a particular functional ornon-functional aspect of the software application under test. A “virtualuser,” as used herein, refers to a virtual representation of the userwhose best user experience will be represented by the virtual user.“Virtual representation,” as used herein, refers to symbolizing a realuser via the use of the user's best user experience that will be used totest the software application under test by the virtual user.

In one embodiment, testing system 104 tests the software applicationusing the virtual users to determine if the functional andnon-functional aspects of the software application meet the users' bestexperiences as discussed in greater detail below.

A description of the software components of testing system 104 used fortesting a software application using the users' best experiences inconnection with the functional and non-functional aspects of thesoftware application under test is provided below in connection withFIG. 2 . A description of the hardware configuration of testing system104 is provided further below in connection with FIG. 3 .

System 100 is not to be limited in scope to any one particular networkarchitecture. System 100 may include any number of computing devices101, software development systems 102, networks 103 and testing systems104.

A discussion regarding the software components used by testing system104 to perform testing of a software application using the users' bestexperiences in connection with the functional and non-functional aspectsof the software application under test is provided below in connectionwith FIG. 2

FIG. 2 is a diagram of the software components used by testing system104 (FIG. 1 ) to perform testing of a software application using theusers' best experiences in connection with the functional andnon-functional aspects of the software application under test inaccordance with an embodiment of the present disclosure.

Referring to FIG. 2 , in conjunction with FIG. 1 , testing system 104includes a monitor engine 201 configured to monitor users' interactions(interactions of the users of computing devices 101) with the functionaland non-functional aspects of the software applications, such as thesoftware applications being developed by the software developers usingsoftware development system 102. “Functional” aspects of the softwareapplications, as used herein, refer to the requirements that thesoftware must perform. For example, such functional aspects may includethe following types of requirements: transaction handling, businessrules, certification requirements, reporting requirements,administrative functions, authorization levels, audit tracking, externalinterfaces, etc.

“Non-functional” aspects of the software applications, as used herein,specify how the software application achieves such requirements. Forexample, the non-functional aspects specify quality attributes of thesoftware, scalability, capacity, maintainability, performance (e.g.,response time), portability, security, availability, reliability,recoverability, serviceability, etc.

As discussed above, user interactions with the functional andnon-functional aspects of the software applications are monitored bymonitor engine 201. Examples of such user interactions includeinteracting with a window, a menu, an icon, a widget, a table, a cursor,a pointer, an insertion point, etc. by the user (e.g., user of computingdevice 101). Data collected from such monitored user interactions mayinclude information, such as the speed in which the table was populated,the number of steps performed in resizing a window or selecting anobject from a menu, the amount of time it takes a user to move thecursor or pointer from one position to another position using usercontrols, such as a mouse, touch pad or similar input device, the amountof time it takes a user to identify the insertion point, such as thelocation at which text is inserted, etc.

In one embodiment, monitor engine 201 utilizes user behavior analyticsin connection with monitoring users' interactions with the functionaland non-functional aspects of the software applications. “User behavioranalytics,” as used herein, refers to collecting, combining andanalyzing quantitative and qualitative user data to understand how usersinteract with the functional and non-functional aspects of the softwareproduct and why. In one embodiment, such user data may be collected bymonitor engine 201 via session recordings, which are renderings of realuser engagement, such as clicks, taps and scrolling, which can bereviewed by an expert to learn the user's experience in connection withinteracting with the functional and non-functional aspects of thesoftware product, such as the time duration in moving the cursor fiveinches across the computer screen. In one embodiment, session recordingoccurs by installing an application, such as on computing device 101, torecord the computer screen when the user (e.g., user of computing device101) is interacting with a functional or non-functional aspect of thesoftware product. Such a session recording is then sent to monitorengine 201 of testing system 104. In one embodiment, the sessionrecording is analyzed by an expert to identify the user's interactionswith the functional and non-functional aspects of the softwareapplication.

In one embodiment, heat maps may be utilized by monitor engine 201 toidentify the user's interactions with the functional and non-functionalaspects of the software product, such as where the user moves the mouseor where the user clicks (or in the case of a mobile device, where theuser taps, swipes or scrolls), as well as information about suchinteractions, such as a deeper color to indicate a longer interaction(e.g., longer time movement of the mouse).

In another embodiment, surveys, such as onsite surveys, may be utilizedby monitor engine 201 to collect personal responses from users (e.g.,users of computing devices 101) as to what functional or non-functionalaspects of the software product are working for them and which are not.In one embodiment, monitor engine 201 utilizes feedback widgets toobtain feedback regarding the user's experience in connection with thefunctional and non-functional aspects of the software product, such asaspects they liked and did not like as well as why such aspects wereliked and not liked. In one embodiment, feedback widgets are installedon computing device 101 to enable the user of computing device 101 toprovide feedback regarding the user's experience in connection with thefunctional and non-functional aspects of the software product. Suchfeedback is then provided to monitor engine 201. “Manual textualfeedback,” as used herein, includes feedback from such widgets as wellas feedback from surveys.

Examples of software tools utilized by monitor engine 201 to obtain suchuser behavior analytics include, but not limited to, Mixpanel™,Amplitude®, Heap®, FullStory®, Sherlock, Tableau®, Dynatrace®, etc.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via log collection and analysis. Log analysis is theprocess of reviewing, interpreting and understanding log files. In oneembodiment, log files may be generated on computing device 101 thatcontain information about the user's interactions with the functionaland non-functional aspects of the software applications, such assoftware applications developed by software developers using softwaredevelopment system 102. In one embodiment, log files are streamed fromcomputing device 101 to monitor engine 201 or they may be stored infiles on computing device 101 for later review by monitor engine 201. Inone embodiment, monitor engine 201 utilizes various software tools(e.g., Sematext Logs, SolarWinds® Loggly, Splunk, Logentries, SumoLogic®, ManageEngine® EventLog Analyzer, Papertrail, etc.) formonitoring users' interactions with the functional and non-functionalaspects of the software applications via log collection and analysis.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via file/screenshot capturing. A “screenshot” captureswhatever is displayed on the screen, such as the screen of computingdevice 101, and saves it in an image file. The image file can betransmitted from computing device 101 to monitor engine 201 foranalysis. In one embodiment, the screenshot is analyzed by an expert toidentify the user's interactions with the functional and non-functionalaspects of the software application.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via a biometric data feed from a wearable device worn bythe user of computing device 101, where in one embodiment, computingdevice 101 represents the wearable device worn by the user. In oneembodiment, the biometric data feed includes personal data resultingfrom the interactions of the functional and/or non-functional aspects ofthe software application relating to the physical, physiological orbehavioral characteristics of individuals. In one embodiment, machinelearning may be utilized to determine the “best” user experience (themost effective or desirable interaction with a functional ornon-functional aspect of the software application, such as undergoing aninteraction with a functional or non-functional aspect of the softwareapplication in the least amount of time) based on the user's physical,physiological or behavioral characteristics after the users indicate,such as via a feedback widget, when a best user experience occurs. Atthe time that the best user experience occurs, the physical,physiological or behavioral characteristics of the user are identified.Over time, a correlation between the physical, physiological orbehavioral characteristics of the user and the best user experiences areidentified.

In one embodiment, monitor engine 201 uses a machine learning algorithm(e.g., supervised learning) to build a mathematical model based onsample data consisting of physical, physiological or behavioralcharacteristics of users in connection with the users' best experiences.Such data may be obtained from biometric data feeds as discussed above.Such a data set is referred to herein as the “training data” which isused by the machine learning algorithm to make predictions or decisionswithout being explicitly programmed to perform the task. In oneembodiment, the training data consists of physical, physiological orbehavioral characteristics of users in connection with the users' bestexperiences. The algorithm iteratively makes predictions on the trainingdata as to the users' best experiences in connection with the physical,physiological or behavioral characteristics of the users. Examples ofsuch supervised learning algorithms include nearest neighbor, NaïveBayes, decision trees, linear regression, support vector machines andneural networks.

In one embodiment, the mathematical model (machine learning model)corresponds to a classification model trained to predict the users' bestexperiences in connection with the physical, physiological or behavioralcharacteristics of the users.

Testing system 104 further includes a profile generator 202 configuredto generate “user experience profiles” for the monitored users (e.g.,users of computing devices 101). A “user experience” profile, as usedherein, refers to a profile that includes data pertaining to a user'sperception of utility, ease of use and efficiency in interacting withthe functional and non-functional aspects of the software applications.In one embodiment, such data may be obtained from the monitored users'interactions by monitor engine 201. For example, such data may includethe user's experience in connection with interacting with a window, amenu, an icon, a widget, a table, a cursor, a pointer, an insertionpoint, etc. Such data may include information, such as the speed inwhich the table was populated, the number of steps performed in resizinga window or selecting an object from a menu, the amount of time it takesa user to move the cursor or pointer from one position to anotherposition using user controls, such as a mouse, touch pad or similarinput device, the amount of time it takes a user to identify theinsertion point, such as the location at which text is inserted, etc.

In one embodiment, user experience profiles correspond to datastructures, which are stored in a storage device (e.g., memory, diskunit) of testing system 104. In one embodiment, such user experienceprofiles include functional and non-functional aspects (e.g., window,widget, menu, table) along with the user's experience (e.g., timeduration, number of steps) in connection with interacting with suchfunctional and non-functional aspects.

In one embodiment, user experience profiles include the specificfunctional and non-functional aspects (e.g., pull-down menu) of thesoftware application as well as the interaction data with respect tosuch an aspect (e.g., speed at which the pull-down menu containing tenentries was populated, number of steps to select an entry in thepull-down menu, etc.), where such information was obtained by monitorengine 201.

In one embodiment, profile generator 202 generates a user experienceprofile for each particular user (e.g., user of computing device 101)whose interactions were monitored by monitor engine 201.

Furthermore, testing system 104 includes a profile examiner 203configured to identify the best user experience out of the generateduser experience profiles in connection with specific functional andnon-functional aspects of the software applications (e.g., populating adrop-down menu), whose interactions thereof by the users (e.g., users ofcomputing devices 101) were monitored by monitor engine 201. That is,profile examiner 203 examines the user experience profiles generated byprofile generator 202 to identify the best user experience in connectionwith specific functional and non-functional aspects of the softwareapplications. For example, out of the user experience profiles examinedby profile examiner 203, it may be determined that a drop-down menu often entries was populated at the fastest time of 1 millisecond. As aresult, it may be deemed that the best user experience corresponds topopulating a drop-down menu of ten entries at 1 millisecond. A “bestuser experience,” as used herein, refers to the most effective ordesirable interaction with a functional or non-functional aspect of thesoftware application, such as undergoing an interaction with afunctional or non-functional aspect of the software application in theleast amount of time.

In one embodiment, profile examiner 203 utilizes natural languageprocessing to identify a functional or non-functional aspect discussedin the user experience profile. In one embodiment, such functional ornon-functional aspects are identified by identifying keywords (e.g.,pull-down menu) that correspond to such functional or non-functionalaspects, where such keywords (e.g., widget) are listed in a datastructure (e.g., table). In one embodiment, such a data structure isstored in a storage device (e.g., memory, disk unit) of testing system104.

In one embodiment, profile examiner 203 identifies the user experienceassociated with such identified functional or non-functional aspectsbased on identifying the data (e.g., time, number of steps) associatedwith such functional and non-functional aspects stored in the userexperience profiles. For example, in a user's experience profile, thenon-functional aspect of populating a pull-down menu of ten items may beassociated with the user's experience in connection with interactingwith such a non-functional aspect (e.g., 1 millisecond to populate thepull-down menu of ten items).

In one embodiment, for each unique functional or non-functional aspectof a software application identified by profile examiner 203, profileexaminer 203 identifies all of the users' experiences in connection withinteracting with such a functional or non-functional aspect of asoftware application based on examining all of the user experienceprofiles generated by profile generator 202. Out of these identifiedusers' experiences, the “best” user experience is identified, such ascorresponding to the shortest time or the least number of steps that areused in connection with interacting with such a functional ornon-functional aspect of the software application. In one embodiment,the determination (e.g., least number of steps, shortest time) for whatconstitutes the “best” user experience in connection with a functionalor non-functional aspect of the software application is determined by anexpert. In one embodiment, profile examiner 203 determines the “best”user experience based on identifying the determining factor (e.g., leastnumber of steps, shortest time) for identifying the “best” userexperience in connection with a particular functional or non-functionalaspect of the software application, where such a determining factor isobtained from a data structure (e.g., table) established by an expert.For example, an expert may indicate in the data structure that thedetermining factor for identifying the “best” user experience inconnection with the non-functional aspect of populating a pull-down menuis the least amount of time. Such information may be obtained from thedata structure by profile examiner 203 using natural language processingto identify the non-functional aspect (e.g., pull-down menu) as well asthe corresponding determining factor for identifying the “best” userexperience. In one embodiment, such a data structure is stored in thestorage device (e.g., memory, disk unit) of testing system 104.

In one embodiment, profile examiner 203 utilizes software tools, such asAggregate Profiler, Atlan®, IBM® InfoSphere® Information Analyser,Informatica® Data Explorer, Melissa® Data Profiler, Microsoft® DOCS, SAPBODS, SAS® DataFlux, Talend® Open Studio, etc., to examine/analyze userexperience profiles as discussed above.

Testing system 104 further includes a virtual user generator 204configured to create virtual users having different types of best userexperiences (obtained from the best user experiences identified byprofile examiner 203) in connection with interacting with differentfunctional and non-functional aspects of the software applications.

A “virtual user,” as used herein, refers to a virtual representation ofthe user whose best user experience will be represented by the virtualuser. “Virtual representation,” as used herein, refers to symbolizing areal user via the use of the user's best user experience that will beused to test the software application under test by the virtual user.

For example, virtual user generator 204 may create a virtual user totest a software application to determine if the software applicationsatisfies the best user experience in connection with the non-functionalaspect of populating a menu of ten items, where such a best userexperience is associated with such a created virtual user. In oneembodiment, such a best user experience is obtained from the best userexperiences identified by profile examiner 203.

In one embodiment, virtual user generator 204 creates a virtual user foreach type of best user experience in connection with each functional ornon-functional aspect of the software application to be tested. In oneembodiment, such best user experiences are obtained from the best userexperiences identified by profile examiner 203.

In one embodiment, such virtual users correspond to profiles withinstructions for testing the software application to determine if itsatisfies the best user experience in connection with each functional ornon-functional aspect of the software application to be tested. In oneembodiment, virtual user generator 204 utilizes a software tool (e.g.,WAPT Pro, Oracle® functional testing visual scripts, OpDesk, etc.) tocreate such profiles.

Testing system 104 further includes a testing engine 205 configured totest the software application using the virtual users to determine ifthe functional and non-functional aspects of the software applicationmeet the users' best experiences. For example, testing engine 205determines if each of the virtual users' best user's experience issatisfied by the software application under test. For instance, theinstructions of the virtual user, such as in a profile (“virtual userprofile”), may be utilized by testing engine 205 to run a test of thefunctional or non-functional aspect of the software application inquestion (the functional or non-functional aspect associated with thevirtual user) to determine if it meets the best user's experience of thevirtual user. In one embodiment, each of the virtual users associatedwith different types of functional and non-functional aspects of thesoftware application are utilized to perform the testing of the softwareapplication to determine if the software application meets the bestuser's experience for each of these functional and non-functionalaspects. In one embodiment, such virtual user profiles are stored in astorage device (e.g., memory, disk unit) of testing unit 104.

In one embodiment, testing engine 205 identifies any functional and/ornon-functional aspect of the software application that fails to satisfya virtual user's best user's experience. For instance, if the user'sbest experience as captured by the virtual user is to populate apull-down menu with ten items in 1 millisecond, and it took the softwareapplication under test 2 milliseconds to populate the same pull-downmenu, then such a failure will be identified by testing engine 205. Inone embodiment, as discussed above, such instructions to populate thesame pull-down menu in the software application under test will beprovided in the virtual user profile.

In one embodiment, testing engine 205 generates a message indicatingwhich functional and/or non-functional aspects of the softwareapplication under test did not satisfy a virtual user's best userexperience. Such a message may be utilized by the software developer tocorrect such deficiencies.

In one embodiment, testing engine 205 utilizes a software tool, such asSelenium®, TestingWhiz, TestComplete®, Ranorex®, Sahi, Watir, ToscaTestsuite®, Katalon Studio®, etc., to test the software applicationusing the virtual users to determine if the functional andnon-functional aspects of the software application meet the users' bestexperiences.

A further description of these and other functions is provided below inconnection with the discussion of the method for testing a softwareapplication, such as during the testing stage of the software productdevelopment process, to determine if the software application satisfiesthe users' best experiences with respect to the functional andnon-functional aspects of the software application.

Prior to the discussion of the method for testing a software applicationto determine if it satisfies the users' best experiences with respect tothe functional and non-functional aspects of the software application, adescription of the hardware configuration of testing system 104 (FIG. 1) is provided below in connection with FIG. 3 .

Referring now to FIG. 3 , FIG. 3 illustrates an embodiment of thepresent disclosure of the hardware configuration of testing system 104(FIG. 1 ) which is representative of a hardware environment forpracticing the present disclosure.

Testing system 104 has a processor 301 connected to various othercomponents by system bus 302. An operating system 303 runs on processor301 and provides control and coordinates the functions of the variouscomponents of FIG. 3 . An application 304 in accordance with theprinciples of the present disclosure runs in conjunction with operatingsystem 303 and provides calls to operating system 303 where the callsimplement the various functions or services to be performed byapplication 304. Application 304 may include, for example, monitorengine 201 (FIG. 2 ), profile generator 202 (FIG. 2 ), profile examiner203 (FIG. 2 ), virtual user generator 204 (FIG. 2 ) and testing engine205 (FIG. 2 ). Furthermore, application 304 may include, for example, aprogram for testing a software application to determine if it satisfiesthe users' best experiences with respect to the functional andnon-functional aspects of the software application as discussed furtherbelow in connection with FIGS. 4-5 .

Referring again to FIG. 3 , read-only memory (“ROM”) 305 is connected tosystem bus 302 and includes a basic input/output system (“BIOS”) thatcontrols certain basic functions of testing system 104. Random accessmemory (“RAM”) 306 and disk adapter 307 are also connected to system bus302. It should be noted that software components including operatingsystem 303 and application 304 may be loaded into RAM 306, which may betesting system's 104 main memory for execution. Disk adapter 307 may bean integrated drive electronics (“IDE”) adapter that communicates with adisk unit 308, e.g., disk drive. It is noted that the program fortesting a software application to determine if it satisfies the users'best experiences with respect to the functional and non-functionalaspects of the software application, as discussed further below inconnection with FIGS. 4-5 , may reside in disk unit 308 or inapplication 304.

Testing system 104 may further include a communications adapter 309connected to bus 302. Communications adapter 309 interconnects bus 302with an outside network (e.g., network 103 of FIG. 1 ) to communicatewith other devices, such as computing devices 101 and softwaredevelopment system 102 of FIG. 1 .

In one embodiment, application 304 of testing system 104 includes thesoftware components of monitor engine 201, profile generator 202,profile examiner 203, virtual user generator 204 and testing engine 205.In one embodiment, such components may be implemented in hardware, wheresuch hardware components would be connected to bus 302. The functionsdiscussed above performed by such components are not generic computerfunctions. As a result, testing system 104 is a particular machine thatis the result of implementing specific, non-generic computer functions.

In one embodiment, the functionality of such software components (e.g.,monitor engine 201, profile generator 202, profile examiner 203, virtualuser generator 204 and testing engine 205) of testing system 104,including the functionality for performing software testing, may beembodied in an application specific integrated circuit.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer, or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be accomplished as one step, executed concurrently,substantially concurrently, in a partially or wholly temporallyoverlapping manner, or the blocks may sometimes be executed in thereverse order, depending upon the functionality involved. It will alsobe noted that each block of the block diagrams and/or flowchartillustration, and combinations of blocks in the block diagrams and/orflowchart illustration, can be implemented by special purposehardware-based systems that perform the specified functions or acts orcarry out combinations of special purpose hardware and computerinstructions.

As stated above, software testing, such as during the testing stage ofthe software product development process, is an investigation conductedto provide stakeholders with information about the quality of thesoftware product or service under test. Software testing can alsoprovide an objective, independent view of the software to allow thebusiness to appreciate and understand the risks of softwareimplementation. Test techniques include the process of executing aprogram or application with the intent of finding failures and verifyingthat the software product is fit for use. Software testing involves theexecution of a software component or system component to evaluate one ormore properties of interest. In general, these properties indicate theextent to which the component or system under test: meets therequirements that guided its design and development, responds correctlyto all kinds of inputs, performs its functions within an acceptabletime, is sufficiently usable, can be installed and run in its intendedenvironments and achieves the general result its stakeholders desire.However, such software testing fails to quantify the users' bestexperiences with the functional and non-functional aspects of thesoftware application under test. For example, when a user gets adrop-down list populated within a millisecond, the user may expect suchan experience in populating another drop-down list in the same or in adifferent application. By not testing software applications to determineif the functional and non-functional aspects of the software applicationsatisfy the users' best experiences, such software applications may notbe well received by the targeted users despite performing well duringthe testing stage of the software product development process due to thepoor user experience of the software application. Unfortunately, thereis not currently a means for testing software applications with theusers' best experiences in connection with the functional andnon-functional aspects of the software application under test.

The embodiments of the present disclosure provide a means for performingsoftware testing that includes testing the software application todetermine if it satisfies the users' best experiences with respect tothe functional and non-functional aspects of the software applicationunder test as discussed below in connection with FIGS. 4 and 5 . FIG. 4is a flowchart of a method for testing a software application todetermine if it satisfies the users' best experiences with respect tothe functional and non-functional aspects of the software application.FIG. 5 illustrates an exemplary user experience profile containinginformation pertaining to the user's experience in connection with thefunctional and/or non-functional aspects of the software application.

As stated above, FIG. 4 is a flowchart of a method 400 for testing asoftware application to determine if it satisfies the users' bestexperiences with respect to the functional and non-functional aspects ofthe software application in accordance with an embodiment of the presentdisclosure.

Referring to FIG. 4 , in conjunction with FIGS. 1-3 , in step 401,monitor engine 201 of testing system 104 monitors users (e.g., users ofcomputing device 101) interacting with functional and non-functionalaspects of software applications, such as software applications beingdeveloped by software developers using software development system 102.

As discussed above, “functional” aspects of the software applications,as used herein, refer to the requirements that the software mustperform. For example, such functional aspects may include the followingtypes of requirements: transaction handling, business rules,certification requirements, reporting requirements, administrativefunctions, authorization levels, audit tracking, external interfaces,etc.

“Non-functional” aspects of the software applications, as used herein,specify how the software application achieves such requirements. Forexample, the non-functional aspects specify quality attributes of thesoftware, scalability, capacity, maintainability, performance (e.g.,response time), portability, security, availability, reliability,recoverability, serviceability, etc.

As also discussed above, user interactions with the functional andnon-functional aspects of the software applications are monitored bymonitor engine 201. Examples of such user interactions includeinteracting with a window, a menu, an icon, a widget, a table, a cursor,a pointer, an insertion point, etc. by the user (e.g., user of computingdevice 101). Data collected from such monitored user interactions mayinclude information, such as the speed in which the table was populated,the number of steps performed in resizing a window or selecting anobject from a menu, the amount of time it takes a user to move thecursor or pointer from one position to another position using usercontrols, such as a mouse, touch pad or similar input device, the amountof time it takes a user to identify the insertion point, such as thelocation at which text is inserted, etc.

In one embodiment, monitor engine 201 utilizes user behavior analyticsin connection with monitoring users' interactions with the functionaland non-functional aspects of the software applications. “User behavioranalytics,” as used herein, refers to collecting, combining andanalyzing quantitative and qualitative user data to understand how usersinteract with the functional and non-functional aspects of the softwareproduct and why. In one embodiment, such user data may be collected bymonitor engine 201 via session recordings, which are renderings of realuser engagement, such as clicks, taps and scrolling, which can bereviewed by an expert to learn the user's experience in connection withinteracting with the functional and non-functional aspects of thesoftware product, such as the time duration in moving the cursor fiveinches across the computer screen. In one embodiment, session recordingoccurs by installing an application, such as on computing device 101, torecord the computer screen when the user (e.g., user of computing device101) is interacting with a functional or non-functional aspect of thesoftware product. Such a session recording is then sent to monitorengine 201 of testing system 104. In one embodiment, the sessionrecording is analyzed by an expert to identify the user's interactionswith the functional and non-functional aspects of the softwareapplication.

In one embodiment, heat maps may be utilized by monitor engine 201 toidentify the user's interactions with the functional and non-functionalaspects of the software product, such as where the user moves the mouseor where the user clicks (or in the case of a mobile device, where theuser taps, swipes or scrolls), as well as information about suchinteractions, such as a deeper color to indicate a longer interaction(e.g., longer time movement of the mouse).

In another embodiment, surveys, such as onsite surveys, may be utilizedby monitor engine 201 to collect personal responses from users (e.g.,users of computing devices 101) as to what functional or non-functionalaspects of the software product are working for them and which are not.In one embodiment, monitor engine 201 utilizes feedback widgets toobtain feedback regarding the user's experience in connection with thefunctional and non-functional aspects of the software product, such asaspects they liked and did not like as well as why such aspects wereliked and not liked. In one embodiment, feedback widgets are installedon computing device 101 to enable the user of computing device 101 toprovide feedback regarding the user's experience in connection with thefunctional and non-functional aspects of the software product. Suchfeedback is then provided to monitor engine 201. “Manual textualfeedback,” as used herein, includes feedback from such widgets as wellas feedback from surveys.

Examples of software tools utilized by monitor engine 201 to obtain suchuser behavior analytics include, but not limited to, Mixpanel™,Amplitude®, Heap®, FullStory®, Sherlock, Tableau®, Dynatrace®, etc.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via log collection and analysis. Log analysis is theprocess of reviewing, interpreting and understanding log files. In oneembodiment, log files may be generated on computing device 101 thatcontain information about the user's interactions with the functionaland non-functional aspects of the software applications, such assoftware applications developed by software developers using softwaredevelopment system 102. In one embodiment, log files are streamed fromcomputing device 101 to monitor engine 201 or they may be stored infiles on computing device 101 for later review by monitor engine 201. Inone embodiment, monitor engine 201 utilizes various software tools(e.g., Sematext Logs, SolarWinds® Loggly, Splunk, Logentries, SumoLogic®, ManageEngine® EventLog Analyzer, Papertrail, etc.) formonitoring users' interactions with the functional and non-functionalaspects of the software applications via log collection and analysis.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via file/screenshot capturing. A “screenshot” captureswhatever is displayed on the screen, such as the screen of computingdevice 101, and saves it in an image file. The image file can betransmitted from computing device 101 to monitor engine 201 foranalysis. In one embodiment, the screenshot is analyzed by an expert toidentify the user's interactions with the functional and non-functionalaspects of the software application.

In another embodiment, monitor engine 201 monitors users' interactionswith the functional and non-functional aspects of the softwareapplications via a biometric data feed from a wearable device worn bythe user of computing device 101, where in one embodiment, computingdevice 101 represents the wearable device worn by the user. In oneembodiment, the biometric data feed includes personal data resultingfrom the interactions of the functional and/or non-functional aspects ofthe software application relating to the physical, physiological orbehavioral characteristics of individuals. In one embodiment, machinelearning may be utilized to determine the “best” user experience (themost effective or desirable interaction with a functional ornon-functional aspect of the software application, such as undergoing aninteraction with a functional or non-functional aspect of the softwareapplication in the least amount of time) based on the user's physical,physiological or behavioral characteristics after the users indicate,such as via a feedback widget, when a best user experience occurs. Atthe time that the best user experience occurs, the physical,physiological or behavioral characteristics of the user are identified.Over time, a correlation between the physical, physiological orbehavioral characteristics of the user and the best user experiences areidentified.

In one embodiment, monitor engine 201 uses a machine learning algorithm(e.g., supervised learning) to build a mathematical model based onsample data consisting of physical, physiological or behavioralcharacteristics of users in connection with the users' best experiences.Such data may be obtained from biometric data feeds as discussed above.Such a data set is referred to herein as the “training data” which isused by the machine learning algorithm to make predictions or decisionswithout being explicitly programmed to perform the task. In oneembodiment, the training data consists of physical, physiological orbehavioral characteristics of users in connection with the users' bestexperiences. The algorithm iteratively makes predictions on the trainingdata as to the users' best experiences in connection with the physical,physiological or behavioral characteristics of the users. Examples ofsuch supervised learning algorithms include nearest neighbor, NaïveBayes, decision trees, linear regression, support vector machines andneural networks.

In one embodiment, the mathematical model (machine learning model)corresponds to a classification model trained to predict the users' bestexperiences in connection with the physical, physiological or behavioralcharacteristics of the users.

In step 402, profile generator 202 of testing system 104 generates userexperience profiles for the monitored users based on the monitoredinteractions.

As discussed above, a “user experience” profile, as used herein, refersto a profile that includes data pertaining to a user's perception ofutility, ease of use and efficiency in interacting with the functionaland non-functional aspects of the software applications. In oneembodiment, such data may be obtained from the monitored users'interactions by monitor engine 201. For example, such data may includethe user's experience in connection with interacting with a window, amenu, an icon, a widget, a table, a cursor, a pointer, an insertionpoint, etc. Such data may include information, such as the speed inwhich the table was populated, the number of steps performed in resizinga window or selecting an object from a menu, the amount of time it takesa user to move the cursor or pointer from one position to anotherposition using user controls, such as a mouse, touch pad or similarinput device, the amount of time it takes a user to identify theinsertion point, such as the location at which text is inserted, etc.

In one embodiment, user experience profiles correspond to datastructures, which are stored in a storage device (e.g., memory 305, diskunit 308) of testing system 104. In one embodiment, such user experienceprofiles include functional and non-functional aspects (e.g., window,widget, menu, table) along with the user's experience (e.g., timeduration, number of steps) in connection with interacting with suchfunctional and non-functional aspects.

In one embodiment, user experience profiles include the specificfunctional and non-functional aspects (e.g., pull-down menu) of thesoftware application as well as the interaction data with respect tosuch an aspect (e.g., speed at which the pull-down menu containing tenentries was populated, number of steps to select an entry in thepull-down menu, etc.), where such information was obtained by monitorengine 201.

In one embodiment, profile generator 202 generates a user experienceprofile for each particular user (e.g., user of computing device 101)whose interactions were monitored by monitor engine 201.

An illustration of an exemplary user experience profile containinginformation pertaining to the user's experience in connection with thefunctional and/or non-functional aspects of the software application isdiscussed below in connection with FIG. 5 .

Referring to FIG. 5 , FIG. 5 illustrates an exemplary user experienceprofile containing information pertaining to the user's experience inconnection with the functional and/or non-functional aspects of thesoftware application in accordance with an embodiment of the presentdisclosure.

As shown in FIG. 5 , user experience profile 500 includes a listing ofthe functional and/or non-functional aspects 501 along with thecorresponding user experience 502. For example, the non-functionalaspect of populating a pull-down menu with ten items has a correspondinguser experience of 1.12 milliseconds. In another example, thenon-functional aspect of resizing a table with twenty items has acorresponding user experience of 2.12 milliseconds. In a furtherexample, the non-functional aspect of selecting an entry from thepull-down menu has a corresponding user experience of 3 steps.

Returning to FIG. 4 , in conjunction with FIGS. 1-3 and 5 , in step 403,profile examiner 203 of testing system 104 identifies the best userexperience in connection with interacting with specific functional andnon-functional aspects of the software applications based on thegenerated user experience profiles.

As stated above, profile examiner 203 is configured to identify the bestuser experience out of the generated user experience profiles inconnection with specific functional and non-functional aspects of thesoftware applications (e.g., populating a drop-down menu), whoseinteractions thereof by the users (e.g., users of computing devices 101)were monitored by monitor engine 201. That is, profile examiner 203examines the user experience profiles generated by profile generator 202to identify the best user experience in connection with specificfunctional and non-functional aspects of the software applications. Forexample, out of the user experience profiles examined by profileexaminer 203, it may be determined that a drop-down menu of ten entrieswas populated at the fastest time of 1 millisecond. As a result, it maybe deemed that the best user experience corresponds to populating adrop-down menu of ten entries at 1 millisecond. A “best userexperience,” as used herein, refers to the most effective or desirableinteraction with a functional or non-functional aspect of the softwareapplication, such as undergoing an interaction with a functional ornon-functional aspect of the software application in the least amount oftime.

In one embodiment, profile examiner 203 utilizes natural languageprocessing to identify a functional or non-functional aspect discussedin the user experience profile. In one embodiment, such functional ornon-functional aspects are identified by identifying keywords (e.g.,pull-down menu) that correspond to such functional or non-functionalaspects, where such keywords (e.g., widget) are listed in a datastructure (e.g., table). In one embodiment, such a data structure isstored in a storage device (e.g., memory 305, disk unit 308) of testingsystem 104.

In one embodiment, profile examiner 203 identifies the user experienceassociated with such identified functional or non-functional aspectsbased on identifying the data (e.g., time, number of steps) associatedwith such functional and non-functional aspects stored in the userexperience profiles. For example, in a user's experience profile, thenon-functional aspect of populating a pull-down menu of ten items may beassociated with the user's experience in connection with interactingwith such a non-functional aspect (e.g., 1 millisecond to populate thepull-down menu of ten items).

In one embodiment, for each unique functional or non-functional aspectof a software application identified by profile examiner 203, profileexaminer 203 identifies all of the users' experiences in connection withinteracting with such a functional or non-functional aspect of asoftware application based on examining all of the user experienceprofiles generated by profile generator 202. Out of these identifiedusers' experiences, the “best” user experience is identified, such ascorresponding to the shortest time or the least number of steps that areused in connection with interacting with such a functional ornon-functional aspect of the software application. In one embodiment,the determination (e.g., least number of steps, shortest time) for whatconstitutes the “best” user experience in connection with a functionalor non-functional aspect of the software application is determined by anexpert. In one embodiment, profile examiner 203 determines the “best”user experience based on identifying the determining factor (e.g., leastnumber of steps, shortest time) for identifying the “best” userexperience in connection with a particular functional or non-functionalaspect of the software application, where such a determining factor isobtained from a data structure (e.g., table) established by an expert.For example, an expert may indicate in the data structure that thedetermining factor for identifying the “best” user experience inconnection with the non-functional aspect of populating a pull-down menuis the least amount of time. Such information may be obtained from thedata structure by profile examiner 203 using natural language processingto identify the non-functional aspect (e.g., pull-down menu) as well asthe corresponding determining factor for identifying the “best” userexperience. In one embodiment, such a data structure is stored in thestorage device (e.g., memory 305, disk unit 308) of testing system 104.

In one embodiment, profile examiner 203 utilizes software tools, such asAggregate Profiler, Atlan®, IBM® InfoSphere® Information Analyser,Informatica® Data Explorer, Melissa® Data Profiler, Microsoft® DOCS, SAPBODS, SAS® DataFlux, Talend® Open Studio, etc., to examine/analyze userexperience profiles as discussed above.

In step 404, virtual user generator 204 of testing system 104 createsvirtual users having different types of best user experiences (obtainedfrom the identified best user experiences) in connection withinteracting with different functional and non-functional aspects of thesoftware applications.

As discussed above, a “virtual user,” as used herein, refers to avirtual representation of the user whose best user experience will berepresented by the virtual user. “Virtual representation,” as usedherein, refers to symbolizing a real user via the use of the user's bestuser experience that will be used to test the software application undertest by the virtual user.

For example, virtual user generator 204 may create a virtual user totest a software application to determine if the software applicationsatisfies the best user experience in connection with the non-functionalaspect of populating a menu of ten items, where such a best userexperience is associated with such a created virtual user. In oneembodiment, such a best user experience is obtained from the best userexperiences identified by profile examiner 203.

In one embodiment, virtual user generator 204 creates a virtual user foreach type of best user experience in connection with each functional ornon-functional aspect of the software application to be tested. In oneembodiment, such best user experiences are obtained from the best userexperiences identified by profile examiner 203.

In one embodiment, such virtual users correspond to profiles withinstructions for testing the software application to determine if itsatisfies the best user experience in connection with each functional ornon-functional aspect of the software application to be tested. In oneembodiment, virtual user generator 204 utilizes a software tool (e.g.,WAPT Pro, Oracle® functional testing visual scripts, OpDesk, etc.) tocreate such profiles.

In step 405, testing engine 205 of testing system 104 performs thetesting of a software application using the virtual users to determineif the software application meets the users' best experiences.

For example, testing engine 205 determines if each of the virtual users'best user's experience is satisfied by the software application undertest. For instance, the instructions of the virtual user, such as in aprofile (“virtual user profile”), may be utilized by testing engine 205to run a test of the functional or non-functional aspect of the softwareapplication in question (the functional or non-functional aspectassociated with the virtual user) to determine if it meets the bestuser's experience of the virtual user. In one embodiment, each of thevirtual users associated with different types of functional andnon-functional aspects of the software application are utilized toperform the testing of the software application to determine if thesoftware application meets the best user's experience for each of thesefunctional and non-functional aspects. In one embodiment, such virtualuser profiles are stored in a storage device (e.g., memory 305, diskunit 308) of testing unit 104.

As discussed above, in one embodiment, testing engine 205 identifies anyfunctional and/or non-functional aspect of the software application thatfails to satisfy a virtual user's best user's experience. For instance,if the user's best experience as captured by the virtual user is topopulate a pull-down menu with ten items in 1 millisecond, and it tookthe software application under test 2 milliseconds to populate the samepull-down menu, then such a failure will be identified by testing engine205. In one embodiment, as discussed above, such instructions topopulate the same pull-down menu in the software application under testwill be provided in the virtual user profile.

In one embodiment, testing engine 205 utilizes a software tool, such asSelenium®, TestingWhiz, TestComplete®, Ranorex®, Sahi, Watir, ToscaTestsuite®, Katalon Studio®, etc., to test the software applicationusing the virtual users to determine if the functional andnon-functional aspects of the software application meet the users' bestexperiences.

In step 406, a determination is made by testing engine 205 of testingsystem 104 as to whether each of the virtual users' best user experienceis satisfied by the software application under test.

If each of the virtual users' best user experience is satisfied by thesoftware application under test, then, in step 407, testing engine 205of testing system 104 generates a message indicating that the softwareapplication meets the virtual users' best user experiences. In oneembodiment, such a message may be in the form of a text message, anemail message, an automated telephone call, etc., that is used tocontact a user, such as a software developer.

If, however, each of the virtual users' best user experience was notsatisfied by the software application under test, then, in step 408,testing engine 205 of testing system 104 generates a message indicatingwhich functional and/or non-functional aspects of the softwareapplication under test did not satisfy a virtual user's best userexperience. In one embodiment, such a message may be in the form of atext message, an email message, an automated telephone call, etc., thatis used to contact a user, such as a software developer. In oneembodiment, such a message may be utilized by the software developer tocorrect such deficiencies.

In this manner, software applications may be tested to determine if thesoftware applications satisfy the users' best experiences with respectto the functional and non-functional aspects of the software applicationunder test.

Furthermore, the principles of the present disclosure improve thetechnology or technical field involving software testing. As discussedabove, software testing, such as during the testing stage of thesoftware product development process, is an investigation conducted toprovide stakeholders with information about the quality of the softwareproduct or service under test. Software testing can also provide anobjective, independent view of the software to allow the business toappreciate and understand the risks of software implementation. Testtechniques include the process of executing a program or applicationwith the intent of finding failures and verifying that the softwareproduct is fit for use. Software testing involves the execution of asoftware component or system component to evaluate one or moreproperties of interest. In general, these properties indicate the extentto which the component or system under test: meets the requirements thatguided its design and development, responds correctly to all kinds ofinputs, performs its functions within an acceptable time, issufficiently usable, can be installed and run in its intendedenvironments and achieves the general result its stakeholders desire.However, such software testing fails to quantify the users' bestexperiences with the functional and non-functional aspects of thesoftware application under test. For example, when a user gets adrop-down list populated within a millisecond, the user may expect suchan experience in populating another drop-down list in the same or in adifferent application. By not testing software applications to determineif the functional and non-functional aspects of the software applicationsatisfy the users' best experiences, such software applications may notbe well received by the targeted users despite performing well duringthe testing stage of the software product development process due to thepoor user experience of the software application. Unfortunately, thereis not currently a means for testing software applications with theusers' best experiences in connection with the functional andnon-functional aspects of the software application under test.

Embodiments of the present disclosure improve such technology bymonitoring user interactions with the functional and non-functionalaspects of the software applications. “Functional” aspects of thesoftware applications, as used herein, refer to the requirements thatthe software must perform. For example, such functional aspects mayinclude the following types of requirements: transaction handling,business rules, certification requirements, reporting requirements,administrative functions, authorization levels, audit tracking, externalinterfaces, etc. “Non-functional” aspects of the software applications,as used herein, specify how the software application achieves suchrequirements. For example, the non-functional aspects specify qualityattributes of the software, scalability, capacity, maintainability,performance (e.g., response time), portability, security, availability,reliability, recoverability, serviceability, etc. User experienceprofiles are then generated for the monitored users based on themonitored interactions. A “user experience” profile, as used herein,refers to a profile that includes data pertaining to a user's perceptionof utility, ease of use and efficiency in interacting with thefunctional and non-functional aspects of the software applications. Forexample, such data may include the user's experience in connection withinteracting with a window, a menu, an icon, a widget, a table, a cursor,a pointer, an insertion point, etc. The best user experiences inconnection with interacting with specific functional and non-functionalaspects of the software applications are then identified based on thesegenerated user experience profiles. For example, out of the generateduser experience profiles, it may be determined that a drop-down menu often entries was populated at the fastest time of 1 millisecond. As aresult, it may be deemed that the best user experience corresponds topopulating a drop-down menu of ten entries at 1 millisecond. A “bestuser experience,” as used herein, refers to the most effective ordesirable interaction with a functional or non-functional aspect of thesoftware application, such as undergoing an interaction with afunctional or non-functional aspect of the software application in theleast amount of time. Virtual users having different types of best userexperiences (obtained from the identified best user experiences) inconnection with interacting with different functional and non-functionalaspects of the software applications are created. A “virtual user,” asused herein, refers to a virtual representation of the user whose bestuser experience will be represented by the virtual user. “Virtualrepresentation,” as used herein, refers to symbolizing a real user viathe use of the user's best user experience that will be used to test thesoftware application under test by the virtual user. For example, avirtual user is created to test a software application to determine ifthe software application satisfies the best user experience inconnection with the non-functional aspect of populating a menu of tenitems, where such a best user experience is associated with such acreated virtual user. The software application is then tested using thevirtual users to determine if the functional and non-functional aspectsof the software application satisfy the best user experiences associatedwith the virtual users. If some of the functional or non-functionalaspects of the software application did not satisfy the best userexperiences associated with the virtual users, then a message may begenerated indicating which functional and/or non-functional aspects ofthe software application under test did not satisfy a virtual user'sbest user experience. Such a message may be utilized by the softwaredeveloper to correct such deficiencies. In this manner, softwareapplications may be tested to determine if the software applicationssatisfy the users' best experiences with respect to the functional andnon-functional aspects of the software application under test.Furthermore, in this manner, there is an improvement in the technicalfield involving software testing.

The technical solution provided by the present disclosure cannot beperformed in the human mind or by a human using a pen and paper. Thatis, the technical solution provided by the present disclosure could notbe accomplished in the human mind or by a human using a pen and paper inany reasonable amount of time and with any reasonable expectation ofaccuracy without the use of a computer.

The descriptions of the various embodiments of the present disclosurehave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1. A computer-implemented method for testing a software application, themethod comprising: identifying best user experiences in connection withinteracting with specific functional and non-functional aspects ofsoftware applications; creating virtual users having different types ofbest user experiences, which are obtained from said identified best userexperiences, in connection with interacting with different functionaland non-functional aspects of said software applications; and performingtesting of said software application using said virtual users todetermine if functional and non-functional aspects of said softwareapplication satisfy said best user experiences associated with saidvirtual users.
 2. The method as recited in claim 1, wherein each of saidcreated virtual users is utilized to determine if a specific functionalor non-functional aspect of said software application meets a best userexperience associated with said specific functional or non-functionalaspect of said software application.
 3. The method as recited in claim 1further comprising: generating a message indicating that said softwareapplication meets said virtual users' best user experiences in responseto said software application satisfying each of said virtual users' bestuser experience.
 4. The method as recited in claim 1 further comprising:generating a message indicating which functional or non-functionalaspect of said software application did not meet a virtual user's bestuser experience in response to said software application not meetingsaid virtual user's best user experience in connection with saidfunctional or non-functional aspect of said software application.
 5. Themethod as recited in claim 1 further comprising: generating userexperience profiles based on user interactions with functional andnon-functional aspects of software applications.
 6. The method asrecited in claim 5 further comprising: identifying said best userexperiences in connection with interacting with said specific functionaland non-functional aspects of said software applications based on saidgenerated user experience profiles.
 7. The method as recited in claim 5,wherein said interactions with said functional and non-functionalaspects of said software applications comprise interactions selectedfrom the group consisting of: a window, a menu, an icon, a widget, atab, a cursor, a pointer, and an insertion point.
 8. A computer programproduct for testing a software application, the computer program productcomprising one or more computer readable storage mediums having programcode embodied therewith, the program code comprising programminginstructions for: identifying best user experiences in connection withinteracting with specific functional and non-functional aspects ofsoftware applications; creating virtual users having different types ofbest user experiences, which are obtained from said identified best userexperiences, in connection with interacting with different functionaland non-functional aspects of said software applications; and performingtesting of said software application using said virtual users todetermine if functional and non-functional aspects of said softwareapplication satisfy said best user experiences associated with saidvirtual users.
 9. The computer program product as recited in claim 8,wherein each of said created virtual users is utilized to determine if aspecific functional or non-functional aspect of said softwareapplication meets a best user experience associated with said specificfunctional or non-functional aspect of said software application. 10.The computer program product as recited in claim 8, wherein the programcode further comprises the programming instructions for: generating amessage indicating that said software application meets said virtualusers' best user experiences in response to said software applicationsatisfying each of said virtual users' best user experience.
 11. Thecomputer program product as recited in claim 8, wherein the program codefurther comprises the programming instructions for: generating a messageindicating which functional or non-functional aspect of said softwareapplication did not meet a virtual user's best user experience inresponse to said software application not meeting said virtual user'sbest user experience in connection with said functional ornon-functional aspect of said software application.
 12. The computerprogram product as recited in claim 8, wherein the program code furthercomprises the programming instructions for: generating user experienceprofiles based on user interactions with functional and non-functionalaspects of software applications.
 13. The computer program product asrecited in claim 12, wherein the program code further comprises theprogramming instructions for: identifying said best user experiences inconnection with interacting with said specific functional andnon-functional aspects of said software applications based on saidgenerated user experience profiles.
 14. The computer program product asrecited in claim 12, wherein said interactions with said functional andnon-functional aspects of said software applications compriseinteractions selected from the group consisting of: a window, a menu, anicon, a widget, a tab, a cursor, a pointer, and an insertion point. 15.A system, comprising: a memory for storing a computer program fortesting a software application; and a processor connected to saidmemory, wherein said processor is configured to execute programinstructions of the computer program comprising: identifying best userexperiences in connection with interacting with specific functional andnon-functional aspects of software applications; creating virtual usershaving different types of best user experiences, which are obtained fromsaid identified best user experiences, in connection with interactingwith different functional and non-functional aspects of said softwareapplications; and performing testing of said software application usingsaid virtual users to determine if functional and non-functional aspectsof said software application satisfy said best user experiencesassociated with said virtual users.
 16. The system as recited in claim15, wherein each of said created virtual users is utilized to determineif a specific functional or non-functional aspect of said softwareapplication meets a best user experience associated with said specificfunctional or non-functional aspect of said software application. 17.The system as recited in claim 15, wherein the program instructions ofthe computer program further comprise: generating a message indicatingthat said software application meets said virtual users' best userexperiences in response to said software application satisfying each ofsaid virtual users' best user experience.
 18. The system as recited inclaim 15, wherein the program instructions of the computer programfurther comprise: generating a message indicating which functional ornon-functional aspect of said software application did not meet avirtual user's best user experience in response to said softwareapplication not meeting said virtual user's best user experience inconnection with said functional or non-functional aspect of saidsoftware application.
 19. The system as recited in claim 15, wherein theprogram instructions of the computer program further comprise:generating user experience profiles based on user interactions withfunctional and non-functional aspects of software applications.
 20. Thesystem as recited in claim 19, wherein the program instructions of thecomputer program further comprise: identifying said best userexperiences in connection with interacting with said specific functionaland non-functional aspects of said software applications based on saidgenerated user experience profiles.